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(about  two-thirds  of  the  nllltary  responses  were  from  Air  Force  Agencies.) 

Project  managers  were  relatively  willing  to  release  all  classes  of  data  except 
direct  dollar  costs,  although  feelings  varied  widely.  All  projects  had  an 
organized  method  of  planning  and  producing  software,  but  few  thought  actual 
planning  was  adequate.  (In  general,  projects  felt  that  they  were  rushed  into 
code  production  without  adequate  designs.)  Koat  projects  endorsed  the  utility 
of  reviews,  but  less  than  half  felt  the  re^tfews  were  adequate.  Most  projects 
did  not  use  military  standard  practlces^Jmt  had  internal  procedures.  Proce- 
dures were  critlsed  for  lack  of  stu^metflzation,  and  inadequate  information. 
Managers  controlled  their  prgjmsfV'through  progress  reports  rather  than  more 
detailed  developmenta^^arfUraation,  and  felt  the  information  they  got  were 
most  lnadoqujgm^40^the  lass  well-structured  areas  of  program  development: 
analys4g»<Ba design.  Bie  data  they  received  was  rated  non-standard,  subjective, 
^^imad'rffquastlonable  accuracy  and  validity. 

oiomation,  standardisation,  systenizatlon  with  full  understanding  of  needs 
and  provisions,  and  indapsndent  verification  through  audit  checks  were  seen 
as  the  a»st  promising  solutions  to  data  collection  problems. 
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SOFTWARE  DEVELOPMENT  DATA  COLLECTION: 

SURVEY  OF  PROJECT  MANAGERS 

1.  INTROOUCTIOH 

The  objective  of  this  survey  was  to  gain  an  assessment  from  first  hand  experi- 
ence regarding  the  problems  associated  with  the  collection  of  software  develop- 
ment data.  When  the  study  of  software  data  collection  problems  was  first 
undertaken  (See  Volume  002  of  this  report),  the  project  members 
Intuitively  expected  the  literature  to  provide  such  an  assessment,  an  Intuition 
that  proved  unfounded.  There  were  actually  very  few  studies  reporting  real 
experience  In  the  collection  of  data  and  not  many  more  speculating  about  them. 
To  fill  this  lack,  the  project  prepared  a questionnaire  covering  the  problems 
of  data  collection  at  these  were  tentatively  revealed  by  our  Initial  Investi- 
gations and  administered  It  to  a sample  of  project  managers  at  SDC  and  at  Pro- 
gram Management  Offices  In  the  military. 

Responses  were  received  from  15  SOC  project  managers  stationed  at  Santa  Monica, 
Colorado  Springs,  Hashlngton,  and  Huntsville.  Responses  were  received  from  10 
(largely  with  tpe*te  answers)  Military  program  management  personnel,  two  thirds 
from  Air  Force  Agencies  (ESO,  ANC,  SAMSO,  SAMTECH)  and  the  remainder  from  Army 
and  Navy  offices.  Response  rates  were  about  30t  from  Internal  SOC  sources  and 
20%  from  military  agencies. 

2.  PROJECT  CHARACTERISTICS 

Most  projects  were  engaged  In  scientific,  engineering  or  RAO  projects  with 
very  few  In  business  data  processing,  as  might  be  reported.  Time  sharing  and 
batch  were  about  equally  employed  with  a great  many  projects  using  both 
and  a few  using  remote  batch  (4  projects  out  of  25  ).  Almost  all  SOC  pro- 
jects reporting  were  engaged  In  federally  sponsored  developments,  but  some  were 
dealing  tdtfi  private  Industry,  others  with  local  government  and  soma  were 


internally  sponsored.  The  computers  used  were  very  heavily  IBM  360/370  or 
HIS  6000  series  machines,  with  a sprinkling  of  CDC,  Xerox,  UNIVAC,  and 
Burroughs  machines,  plus  some  minicomputers  In  communications  and  avionics 
applications.  The  only  specialized  peripherals  widely  Interfaced  with  were 
communication  equipment,  but  four  projects  dealt  with  avionics  and  sensor 
equipment  and  two  with  navigation  gear. 

Project  size  varied  from  2 persons  and  5000  object  Instructions  to  200  persons 
and  1.2  million  Instructions.  Modularization  used  on  the  projects  did  not 
seem  extreme,  ranging  from  around  300  to  2000  object  Instructions  per  module 
or  routine.  Languages  used  include  FORTRAN,  JOVIAL,  COBOL,  GMAP,  BASIC  and 
assembly  language.  Experience  ranged  from  2.5  to  15  years  with  a modal  value 
of  8.  Projects  ranged  from  8 months  to  6 years  In  length. 

Almost  all  projects  used  compilers,  utilities,  dumps,  link  editors  and  pro> 
gram  libraries.  Nearly  as  many  used  debug  packages,  recording/reduction  tools, 
and  flow  charters.  Almost  a third  of  them  used  simulators  and  data  manage- 
ment tools  and  a sprinkling  used  verifiers,  auditors  and  timers.  Half  the 
projects  thought  their  tool  package  quite  adequate,  some  (10X)  thought  they 
were  marginal  and  a few  (15%)  thought  them  insufficient.  Additional  tools 
suggested  Included  project  management  and  scheduling,  configuration  manage- 
ment, autoaatlc  documenters,  and  automatic  verifiers  for  all  languages. 

Graphic  output  and  debugging  packages  that  are  easier  to  use,  less  constraining 
and  having  few  Instrumentation  effects  (I.e.,  whose  use  forces  particular 
structure  on  the  tested  Item)  were  suggested.  Tools  were  also  criticized  for 
being  Inadequately  standardized— compilers  too  diverse  In  the  efficiency  of 
code  produced  and  the  errors  checked,  data  management  programs  were  too 
specialized,  flow  charters  existed  at  several  levels  of  deUII,  and  compliance 
with  specification  standards  was  Judged  very  variable. 

Except  for  the  R(0  projects,  projects  are  of  modurate  size,  produced  by 
experienced  craftsmen  using  traditional  or  standard  tools  and  techniques. 

The  possible  exception  lies  In  the  large  proportion  of  projects  using  Inter- 
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active  programming  at  least  part  of  the  time. 

3.  DA1A  SENSITIVITY 

Data  concerning  software  development  varies  in  sensitivity.  Data  may  either 
reflect  adversely  on  project  performance  or  disclose  proprietary  information 
to  competitors.  Managers  may  be  reluctant  to  release  such  data  freely.  How- 
ever, to  be  maximally  useful  for  comparative  studies  of  software  methodology 
and  reliability,  all  types  of  project  data  are  desirable.  In  this  question- 
naire the  respondents  were  asked  to  rate  their  relative  reluctance  or  willing- 
ness to  release  data  to  a semi  public  data  bank  given  reasonable  guarantees  of 
privacy  to  avoid  open  criticism  of  project  performance. 


Unfortunately,  quite  a few  of  the  military  PMO  personnel  did  not  respond  to 
this  section  on  the  apparent  grounds  that  such  reporting  did  not  apply  to  their 
operation.  It  was  intended  that  the  PMO's  rate  their  contractors  in  terms  of 
the  resistance  the  PMO  encountered  in  gathering  data  from  them,  but  either  the 
questionnaire  instructions  failed  to  make  this  clear  or  the  project  management 
offices  did  not  collect  such  data  or  experience  customer  behavior  in  regard  to 
it.  Consequently,  no  separate  analysis  for  PMO  responses  is  reasonable;  the 
ratings  of  those  who  did  respond  are  included  in  the  internal  counts  shown  in 
the  following  figures. 

There  was  a wide  range  of  opinion  on  every  item.  On  every  item  at  least  some- 
one said  that  they  would  be  very  reluctant  and  someone  else  said  they  would  be 

very  willing.  (A  rating  of  '1*  iiMHcetes  willingness,  a rating  of  '5'  indicates' 
reluctance  in  the  figures.) 


Figure  1 shows  cost  and  schedule  data  ratings.  To  summarize,  .*'e  project  man- 
agers  were  most  reluctant  to  releese  dollar  costs,  rather  willing  to  release 
costs  In  mahpower  and  very  wlllinf  to  release  cosn  In  machine  time.  Although 
not  pronounced,  there  Is  soma  greater  reluctance  to  release  fine  costs  (costs 
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per  Instruction  or  per  Individual  activity)  than  coarse  costs  (costs  per  total 
configuration  Item  or  total  task.)  The  managers  »»ere  quite  reluctant  to  re- 
lease variance  data  for  costs  or  schedules  that  made  them  look  bad  (overruns) 
and  while  more  willing  to  release  "good  news"  (underruns),  they  were  not  over- 
whelmingly so.  These  findings  are  quite  In  line  with  what  one  would  expect, 
except  for  the  rather  sharp  differences  showing  up  between  the  "close  to  the 
vest"  and  "let  It  all  hang  out"  attitudes  held  by  different  managers. 

Figure  2 shows  the  relative  willingness  or  reluctance  of  managers  to  release 
resource  utilization  Information.  Various  manpower  breakdowns  received  moder- 
ate Indorsement  (more  favorable  than  unfavorable)  and  computer  time  breakdowns 
was  very  definitely  approved.  (There  were  no  facility  managers  in  the  sample; 
It  might  have  made  a difference.)  Surprisingly,  the  managers  were  quite 
willing  to  release  personnel  turnover  data  whether  that  data  were  for  project 
members  or  for  key  technical  and  managerial  personnel . 

Figure  3 shows  how  the  project  managers  felt  about  releasing  evaluations  of 
project  performance  whether  these  were  proficiency  ratings  of  personnel  or 
computing  facility  efficiency.  Project  managers  were  quite  ambivalent  about 
releasing  personnel  proficiency  ratings— as  can  be  seen  they  scattered  their 
ratings  fairly  uninformally  across  the  spectrum— but  quite  willing  to  release 
experience  figures.  Again,  for  productivity  ratings  they  were  somevAiat 
restrained  about  releasing  work  unit  costs  or  durations,  but  were  reasonably 
willing  to  rate  the  computer  and  other  support  activities  on  their  efficiency. 
Again,  except  for  the  relative  willingness  of  some  managers  to  release  pro- 
ficiency ratings  of  their  project  members,  those  findings  are  fairly  well  In 
line  with  what  one  would  expect. 

Figure  4 Is  also  fairly  well  In  line  with  what  one  would  expect— managers  are 
quite  willing  to  release  configuration  data— unless  evaluative  ratings  of 
efficiency,  reliability,  etc.,  are  Involved.  Even  here  the  managers  are  more 
willing  than  not  to  share  their  experiences. 
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Figure  5 compares  the  sensitivity  of  data  concerning  various  events  that 
might  lead  to  program  changes:  requests  for  modification,  reports  of  problems 
encountered,  and  reports  of  suspected  program  errors.  The  results  are  quite 
In  line  Mith  previous  findings.  Managers  are  quite  willing  to  release  objec- 
tive Information  about  the  numbers,  sizes  and  types  of  changes  encountered,  but 
more  reluctant  to  release  data  on  cost  and  schedule  Impacts.  Again,  the 
division  between  the  willing  and  the  reluctant  Is  quite  plainly  polarized.  No 
Information  Is  readily  available  from  those  two  parties  (the  returns  were  a11 
anonymous,  effectively  prohibiting  follow  up).  However,  a more  penetrating 
Inquiry  might  yield  Interesting  results. 
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4.  SOFTWARE  DEVELOPMENT  PROCESS 

Software  ts  developed  and  managed  in  a variety  of  ways.  Currently,  new  techno- 
logies are  being  tried  and  the  most  cost  effective  and  reliable  methods  have 
not  been  determined.  However,  it  seems  a number  of  project  managers  have 
solved  many  of  the  problems  of  software  development  to  their  own  satisfaction. 
Their  answers  to  several  questions  aimed  at  soliciting  satisfaction  with  the 
suggested  improvements  to  their  current  methodology  can  be  taken  as  the 
criterion. 

4.1  PLANNING 

Almost  all  respondents  stated  that  they  had  an  organized  method  for  producing 
software  for  all  phases  of  development.  Similarly,  plans  were  laid  for  man- 
power utilization,  schedules  and  budgets  for  work,  project  organization, 
financial  matters  and  testing.  Those  with  documentation  plans  and  configura- 
tion management  plans  were  only  slightly  fewer.  More  specialized  plans,  like 
those  for  facilities,  training,  conversion,  support  and  liaison  fell  to 
around  25?  of  the  respondents.  However,  when  asked  to  evaluate  the  adequacy 
of  planning,  only  six  respondents  felt  planning  was  adequate.  Ten  said  out- 
right that  it  was  not  and  four  said  somewhat  or  only  in  special  projects. 
Comments  received  on  this  item  tended  toward  criticizing  product  plans  (i.e., 
designing)  as  being  inadequately  done.  Some  of  the  comments  include: 

"Monetary  constraints  tend  to  drive  projects  into  production 
phase  prior  to  proper  design  completion— customer  is  to  blame." 

"First  major  delivery  is  almost  always  too  early  to  allow 
adequate  system  design  and  project  planning." 

"Usually  t adequate],  although  the  time  required  for  these 
Activities  [development  and  test]  and  product  reliability 
can  both  be  Improved  by  allotting  more  time  to  design." 


"[Planning  could  be  improved  by]  a detailed  work  break- 
down structure." 


4.2  KEY  PERSONNEL 

Approximately  40%  of  the  respondents  said  at  least  some  personnel  were  speci- 
fied by  the  proposal  and/or  the  contract.  Thus  current  practices  do  reflect 
an  emphasis  upon  key  technical  resources. 

4.3  INDEPENDENT  TEST  TEAMS 

One  of  the  proposed  techniques  for  ensuring  greater  reliability  of  software 
Is  the  utilization  of  Independent  test  teams  to  perform  final  design  verifi- 
cation and  validation.  Less  than  a third  of  the  project  managers  sur- 
veyed said  they  used  such  teams.  When  used,  such  test  teams  were  seen  as 
doing  an  effective  job  from  the  customers  point  of  view.  The  few  comments 
made  regarding  these  questions  said: 

"Monitary  constraints  prohibited  this  concept  [on  this 
project].  Consequently,  many  problems  occurred  In 
developing  adequate,  for  the  customer,  test  plans  and 
procedures . " 

"Independent  tests  [and  reviews]  work  best  when  the 
schedule  Is  allowed  to  slip,  to  accommottete  delays  in 
the  review  process." 

"[Independent  test  teams  were]  planned,  but  not  effectively 
applied." 

"Partially [adequate]  from  the  customer's  point  of  view, 
although  a great  part  of  their  effectiveness  derives  from 
the  adequacy  of  the  Part  I specs  and  the  functional  break- 
out of  the  CPCI  [for  the  test  team  to  work  from]." 


4.4 


SOFTWARE  REVIEW  PROCESSES 


The  use  of  software  reviews  to  Increase  the  reliability  of  software,  especially 
during  the  critical  early  period  of  software  development,  has  received  con- 
siderable attention  in  recent  years.  Respondents  were  asked  if  they  did  have 
a set  of  consistent  and  periodic  reviews  and,  if  so,  which  of  the  standard 
military  reviews  had  been  instituted.  Two  thirds  of  the  respondents  said  they 
did  use  systematic  review  procedures.  Even  more  (nearly  80%)  used  Preliminary 
Design  Reviews  (POR's)  and  Critical  Design  Reviews  (COR's).  Only  50%  however, 
indicated  they  used  Performance  Requirement  Reviews  (PRR's)  or  Formal  Qualifi- 
cation Reviews  (FQR's).  To  this  list*  however,  the  military  respondents 
added  System  Requirements  Review,  (SRR)  and  Functional  and  Physical  Configu- 
ration Audits  (FCA  and  PCA),  and  the  project  managers  added  informal  design 
reviews,  structured  walk  throughs,  and  technical  interchange  meetings. 

The  respondents  were  asked  whether  or  not  there  was  a systematic  procedure 
for  incorporating  the  discrepancies  found  in  the  reviews  into  the  software 
development  product  in  a timely  and  cost  effective  way.  They  were  also  asked 
if  the  customer  was  involved  in  the  review  and  discrepancy  resolution  process 
and,  if  not,  would  it  help  if  they  were.  About  40%  of  the  respondents  said 
they  had  such  procedures,  but  the  conments  to  the  question  indicated  that  the 
procedures  varied  from  formal  design  change  and  error  report  processing  to 
informal  day  to  day  Interactions.  About  the  same  proportion,  with  some 
qualifications,  indicated  that  the  customer  was  involved  in  software  reviews 
end  that  his  attendance  was  helpful.  Some  comments  received  were: 

"The  project  had  excellent  customer  involvement;  all 
changes  were  designed,  costed  and  scheduled  practically 
on  a day  to  day  basis.  Customer  personnel  were  on  the 
technical  team  [a  Joint  progranning  project]  and 
participation  was  excellent." 


"[Discrepancies  were  resolved]  by  use  of  Design  Modification 
Requests  and  Discrepancy  and  Correction  Reports  and  at  the 
larger,  more  formal  reviews  by  issuing  Action  Items.  More 
customer  attendance  at  reviews  would  be  desirable." 

"Depends  upon  what  you  call  "systematic".  Discrepancies 
are  certainly  corrected.  More  customer  participation  would 
probably  hinder  the  operation.  Our  customers  are  mainly 
concerned  with  the  final  product  and  not  generally  concerned 
with  intermediate  reviews." 

"Hot  sure  [if  more  participation  would  help].  May  complicate 
things." 

"Our  customer  sits  in  judgment  at  all  design  reviews  and 
has  a seat  on  internal  review  connlttees.  However,  he 
typically  does  not  understand  the  system  and  consequently 
slows  down  the  progress  made." 

"After  a baseline,  standard  ECP  processes  are  used:  Design 
Modification  Requests  to  the  Design  Control  Group  at  any 
time.  Customer  personnel  are  in  house  on  the  project.  Their 
participation  in  design  reviews  is  absolutely  Imperative  to 
avoid  argument  and  delay." 

"Modifications  are  made  to  the  design  document  prior  to  formal 
Implementation  of  the  required  changes.  Customers  attend 
informal  as  well  as  formal  reviews." 

"Customer  participates,  but  Is  usually  unqualified  to  con- 
tribute effectively.  If  the  customer  represents  a user, 
it  would  be  much  more  beneficial  if  he  could  keep  the  user 
constantly  involved.  Buyer  staffs  are  frequently  unqualified 
to  monitor  the  development  of  a system." 
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"The  procedure  followed  Is  very  much  a function  of  the 
seriousness  of  the  discrepancy,  the  extent  of  the  effects, 
the  point  at  which  It  Is  discovered,  and  the  people  In- 
volved. The  customer  Is  Involved  through  regular  written 
reports  and  status  meetings." 

"An  effective  action  Item  system  handles  all  discovered 
discrepancies.  Customer  lives  In  the  building." 

"Appropriate  documentation  Is  generated  prior  to  review 
and  review  comments  are  Incorporated  subsequently  con- 
sistent with  configuration  control  concepts.  The  customer 
Is  Involved  In  the  evaluation  of  Design  Review  Packages 
and  coordination  of  comments  either  at  the  Design  Review 
or  Technical  Interchange  Meeting  and  Is  Involved  further 
^ In  design  review,  technical  Interchanges  and  management 

f reviews  prior  to  the  formal  review.  (This  project  is 

primarily  an  Air  Force  organization  project  with  on-site  con- 
H tractor  assistance.)" 

The  respondents  were  asked  to  check  a list  of  Items  that  might  be  Included  In 
the  review  process.  The  Items  and  the  number  Indicating  Inclusion  were: 

Progress  Reports  86X 

Delivery  * Computer  Schedules  71 

Discrepancy  Report  67 

Project  Summaries  62 

Program  Listings/Documentation  62 

Manpower  Costs  57 

Configuration  Management  Report  48 

Computer  Utilization  Report  43 
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The  respondents  were  asked  whether  the  review  processes  were  effective  In  their 
estimation  for  detecting  and  correcting  significent  design  errors  and  how  they 
might  be  Improved.  Only  40%  of  the  respondents  thought  the  reviews  were 
adequately  effective.  Suggestions  of  the  reasons  for  Ineffectiveness  and 
possibilities  for  Improvement,  with  the  first  three  Items  cited  several  times. 

Included.  ^ insufficiently  detailed  reviews 

t Inadequate  budget  and  schedule  for  review 

• Greater  emphasis  on  review  during  the  design  process 

• Continuing  contact  between  contractor  and  customer 

• Joint  generation  by  contractor  and  customer  of 
specifications  during  a concept/design  phase 

• Pre-design  meetings  to  establish  overall  conventions, 
concepts  and  Interfaces  prior  to  Part  I design 

• Control  by  a control  project  office  over  key  technical 
aspects  manned  by  competent,  responsible  personnel  with 
authority  to  force  correct  design  approach 

• Adequate  time  for  design  and  review 

• Improved  cost  estimating  techniques 

The  emphasis  for  Improved  effectiveness  Is  seemingly  concentrated  on  conception 
prior  to  design,  close  llason  with  the  customer,  and  some  Improvement  In  con- 
figuration management  procedures. 

4.5  CONFIGURATION  CONTROL  PRACTICES 

Configuration  control  Is  the  systematic  evaluation,  coordination,  approval  and 
Implementation  of  a computer  program  and  associated  documentation  after  formal 
Identification  of  its  requirements.  The  military  have  established  various 
standard  practices  for  configuration  control  and  for  software  specification. 

In  response  to  questions  on  this  matter.  Just  half  the  respondents  Indicated 
they  had  followed  such  practices  and  believed  them  an  effective  tool  for 
managing  the  reliability  and  productivity  of  the  software  being  developed. 
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Eleven  projects  were  governed  by  MIL-STO's  483  and  490,  six  by  MIL-STD-480, 
three  by  000  Inst.  4120. 17-M,  two  each  for  MIL-STD-499,  NAVSHIPS  0967-011-0011, 
and  internal  progranmlng  standards,  and  one  each  for  MIL-STD  1521,  MIL-R  83313, 
and  APR  800-14.  Although  the  military  has  devoted  a great  deal  of  effort  to 
configuration  management,  it  is  not  as  widely  practices  as  one  might  hope. 

Only  about  25*  of  the  respondents  thought  the  existing  standards  were  adequate 
for  effective  control.  However,  the  only  comment  received  regarding  the 
standard  practices  was  that  they  are  "Too  vague  and  subject  to  Interpretation 
for  effective  control,  especially  as  to  the  level  of  detail  required  in  the 
specification." 

5.  CURRENT  PRACTICES  IN  DATA  COLLECTION 

The  adequacy  of  the  software  data  collected  is  influenced  by  many  factors 
ranging  from  the  standards  employed  in  determining  the  data  to  collect  to  pro- 
ject reluctance  to  release  sensitive  data.  The  respondents  were  first  asked 
what  standards  they  used  and  what  were  their  major  deficiencies. 

5.1  STANDARD  PRACTICES 

First,  very  few  projects  (between  10  and  30%  for  var^ious  standards)  used  the 
military  standard  practices.  However,  many  did  have  standard  data  that  were 
collected  for  almost  every  aspect  of  software  development.  The  list  of  data 
types  and  the  number  of  projects  reporting  standards  for  collection  are  shotwi 
in  Table  1.  It  would  appear  that  the  major  emphasis  is  upon  progress  and  schedule 
reporting  with  less  than  50%  of  the  projects  using  any  standards  for  reporting 
other  data  items. 
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Table  1 . Data  Item  Standards 

Percent  of  the 

Data  Type  Projects  with  Standards 

Schedule  Variance  76 

Cost  Variance  28 

Project  Progress  90 

Problem  Reports  for: 

1.  Design  38 

2.  Analysis  38 

3.  Integration  43 

4.  Code  and  Check  52 

5.  Installation  43 

6.  Test  43 

7.  Operations  24 

Error  Statistics  14 

Manpower  Utilization  52 

Computer  Utilization  49 

Design  Change  Statistics  38 

Software  Nodule  Statistics  33 
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The  respondents  were  then  asked  what  the  major  deficiencies  were  In  the  stan- 
dards they  used.  The  responses.  In  order  of  their  endorsement,  were: 


e Inadequate  for  comparative  study  across  projects  43% 
e Coarseness  of  measures  38 
e Inadequate  Information  on  project  problems  38 

# Failure  to  apply  to  all  phase  of  development  29 
e Inadequate  Information  on  product  errors  19 
e Excessive  collection  costs  14 
e Inadequate  configuration  Information  14 

• Inadequate  Information  on  design  changes  14 
e Subject  to  Interpretation;  vague  14 


If  the  allegation  that  data  are  Inadequate  for  comparative  study  Is  true, 
developmental  costing  modules  developed  from  them  would  have  limited  utility. 
Further,  close  management  would  be  Inhibited  by  coarse  data.  Inadequate  Infor- 
mation and  missing  Information  for  some  phases.  The  other  deficiencies  are 
not  greatly  endorsed  but  Indicate  some  dissatisfaction  with  current  data 
collection  practices  In  these  areas. 

5.2  MANAGEMENT  CONTROL 

The  project  managers  were  asked  which  of  the  data  classes  they  used  for  managing 
their  projects  and  where  in  the  developmental  process  were  the  major  Inadequacies 
In  the  data  Items.  Again,  usage  of  these  data  was  not  overwhelming  and  very  few 
checked  all  data  classes.  Responses  were: 


e 

Progress  reports 

57* 

e 

Error  reports 

33 

e 

Cost  reports  (Including  manpower  and 

33 

computer  utilization) 

e 

Change  reports 

24 

e 

Problem  reports 

24 

e 

Schedule 

5 

t 

Action  Items 

5 

/ 

i 

a 


a 


20 


Agreement  was  greater  on  where  In  the  developmental  process  the  reports  were 
Inadequate.  Responses  were: 


Design  process 

67* 

System  Analysis  phase 

52 

Program  Production 

33 

Phase  over 

10 

Integration  and  Test 

00 

It  would  appear  that  the  managers  were  most  lacking  1n  information  during  the 
early  formative  and  conceptual  aspects  of  the  developmental  process.  Once  the 
product  was  well  defined,  the  data  collected  was  deemed  reasonably  adequate. 
This  does  suggest  that  some  attention  should  be  directed  at  either  greater 
structuring  of  analysis  and  design  or  that  a different  class  of  data  ought  to 
be  collected.  Perhaps  the  responses  are  merely  an  endorsement  of  the 
I uncertainty  that  everyone  feels  while  the  system  Is  being  conceived,  but 

a lack  of  good  management  control  over  the  early  phases  of  development  does 
seem  a most  likely  reality. 

i 

5.3  COMPARATIVE  METHODOLOGY 

' One  of  the  principle  reasons  for  collecting  software  development  data  Is  to 

derive  measures  for  evaluating  and  comparing  different  progranmlng  techniques, 
methods  and  tools.  The  respondents  were  asked  to  evaluate  currently  collected 
data  and  Indicate  the  chief  Inadequacies  In  the  data  for  this  purpose. 


Their  responses  were: 

e Project  comparabi 1 1 ty  62X 

e Subjectivity  and  bias  33 

• Continuity  through  life  cycle  29 

• Reliability  data  19 

e Cost  data  14 

e Quality  assurance  information  *10 
e Standard  baselines  05 


5.4 


COST  FACTORS 


The  project  manegers  were  asked  If  they  thought  that  state-of-the-art  data 
collection  was  too  costly,  where  the  excessive  cost  factors  lay  and  what 
could  be  done  to  reduce  these  costs. 

In  answer  to  where  the  principle  costs  of  collecting  software  development  data 
lay,  the  project  managers  said: 


Volume  of  data 

43X 

Difficulty  of  measuring 

43 

Number  of  measures 

32 

Report  preparation 

32 

Interference  with  work 

33 

Frequency  of  reporting 

24 

Reduction  of  data 

19 

Lack  of  defined  goal 

05 

Additionally,  there  were  several  comments  to  this  question: 

"The  time  required  of  production  personnel  In  preparing 
reports  Is  too  great,  but  support  personnel  to  do  the 
work  are  too  costly." 

"Due  to  the  lack  of  clear  cut  goals  toward  which  data 
collection  can  be  aimed,  too  much  unappllcable  data  Is 
''ollected." 

"Tile  9reparat1on  and  maintenance  of  the  [management] 
data  base  and  the  generation  of  Cementation  are  both 
too  costly." 

"The  handling  of  problem  and  change  requests  Is  In- 
efficient." 

"Manpower  and  the  expense  of  Indoctrination  [training 
In  data  collection  procedures]  are  excessive." 
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"Verification  of  the  data  Is  a major  cost  factor." 

"Attempting  to  collect  too  fine  grain  data." 

"Attempting  to  let  the  collection  of  data  drive  the  system." 

Costs,  especially  unbudgeted  costs,  are  a source  of  great  concern  to  managers 
although  only  a small  proportion  of  the  respondents  said  costs  were  a major 
problem  In  the  collection  of  data  (fourteen  percent,  see  Section  5.1).  The 
suggestions  for  Improvement  made  by  the  managers  provide  a little  more  Insight. 

These  Include: 

e Automation 
e Simplification 
e Standardization 
e Reduction  of  detail 

e Better  evaluation  of  data  collection  requirements 

Each  of  these  suggestions  received  several  endorsements.  There  were  three 
suggestions  for  automation.  Including  the  monitoring  of  a programming  support 
library  as  a data  collection  technique.  There  were  an  equal  number  of  pleas 
for  simplification  ("make  It  easier")  with  an  attendant  effort  to  sell  the 
benefits  of  data  collection  and  to  explain  the  rationale  and  procedures  to 
the  project  ammbers. 

The  requests  for  simplification  were  also  partially  requests  fOr  standarlzed. 
cross-project  and  cross-discipline  systems  and  procedures  that  coifid  be  under- 
^ stood  by  all.  Another  benefit  foreseen  for  standerdlzod  and  automated  systems 

{ c Is  a reduction  of  threatening  pressure  on  the  progrewmirs..1lot  make  them  feel  i 

they  are  being  consUntly  overseen."  Several  persons  felt  less  data  collectloi 
I might  be  the  solution.  One  person  advocated  a reduction  In  the  detail  of  the 

( data  collected.  Another  said:  "Be  pragmatic.  Measure  only  what  can  be 

I realistically  measured."  Another  summed  It  up:  "The  real  question  Is:  Can 

, the  collection  cost  result  In  savings  «d)1ch  are  greater?  If  not,  the  most 


i 

i 

; ’.V* ; 

-*7.  ^ 


f 


cost  effective  solution  nay  be  sinply  to  "wing  It."" 


In  short,  project  managers  would  like  a standard,  simplified  data  collection 
system,  easy  to  use  and  explain,  that  represents  a minimum  of  Interference 
In  ongoing  work.  Whether  such  a system  can  be  obtained  and  still  provide 
adequate  Information  for  close  management  control  and  methodology  research 
remains  to  be  seen. 

6.  DATA  COLLECTION  PROBIRS 

One  of  the  major  objectives  of  the  questionnaire  was  to  gather  experimental 
data  on  the  problems  associated  with  data  collection.  The  project  managers 
were  asked  to  check  off  what  their  major  problems  were  in  collecting  data. 


The  results  were: 

• Subjectivity  of  estimates  52X 

e Interference  with  project  progress  48 

• IncomparablUty  of  projects  38 

• Project  resistance  33 

• Collection  costs  29 

e Invalid  measures  15 

e Distorted  or  falsified  data  15 

e Non-standard  practices  05 

e Insufficient  Information  for  valid  evaluation  05 

• Knowing  what  to  collect  05 

• Convincing  management  of  the  cost  effective-  05 

ness  of  data  collection  05 

e Time  to  handle  minor  details 


25 


6.1  SUBJECTIVITY 

The  subjectivity  of  measures  can  arise  from  a variety  of  causes  langing  from 
the  alleged  Insubstantiality  of  the  softmare  product  to  personal  bias  on  the 
part  of  those  reporting.  Mhen  asked  what  the  contributing  factors  to  the 
subjectivity  of  data  collection  measures  were,  the  project  managers  said: 


e Continual  change  62X 

e Optimistic  bias  48 

e Failure  to  consider  all  elements  43 

e Lack  of  measurable  performance  33 

• Lack  of  "Instrumentation*  29 

• Insubstantial  product  10 

e Innovativeness  of  process  05 


Obviously,  the  project  managers  do  not  agree  with  the  old  saw  that  the  reasons 
for  unreliable  cost  estimates  and  poor  performance  are  a rapidly  changing 
technology  and  a logical,  non-physical  product.  Instead  they  blame  an  un- 
sUble  environment  and  the  fallibility  of  estimators.  Although  it  Is  not 
likely  that  human  nature  will  change,  certainly  an  Improved  data  collection 
system  should  provide  more  objectivity  and  greater  standardization. 

The  managers  gave  very  few  responses  when  asked  how  the  objectivity  of  measures 
might  be  Improved.  They  did  suggest: 

e Develop  standard  parameters 
e Use  Independent  audit  teams 
e Perform  data  collection  research  comparing  a 
number  of  projects  and  Identify  the  factors 
that  contribute  to  the  unwanted  variance. 

These  are,  of  course,  pertinent  suggestions  and  are  part  of  the  justification 
for  the  Repository. 
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INTERFERENCE  WITH  PROJECT  PROGRESS 

When  asked  »ih1ch  major  factors  caused  data  collection  to 
interfere  with  project  progress,  the  managers  checked: 

e Time  required  to  prepare  reports  S2% 

• Distracting  and  Irksome  33 

• Interference  with  line  of  thought  lo 

There  have  been  serious  arguments  advanced  for  developing  more  objective 
measures  that  could  be  taken  without  involving  analytic  and  programning 
personnel.  However,  in  the  eyes  of  the  managers,  although  data  collection 
takes  time  and  is  Irksome,  it  does  not  actively  interfere  with  the  worker's 
thought  processes. 

Suggestions  for  avoiding  interference  included: 

e Automation 

e Report  at  higher  levels 

e Streamline  and  standardize 

e Use  Independent  audits 

e Place  a data  collector  on  the  project  staff 

Some  managers  laid  It  on  the  line: 

"Some  Interference  Is  bound  to  result  If  you  are  to  get 
to  the  root  of  problems." 

"If  data  collection  Interferes,  you  are  doing  It  wrong." 

In  short,  quite  a few  managers  believe  that  we  are  doing  It  wrong  and  that  a 
more  standardized,  automated  data  collection  system  using  non-1nvo1«ed  data 
collectors  Is  the  way  to  go.  Although  formal  data  collection  and  reporting 
costs  are  quite  small  (estimated  at  3*  of  project  costs,  see  Volume  002), 
they  mey  require  substantially  more  time  of  analysts  and  prograawrs  than  the 
statistic  Indicates.  More  efficient  collection  procedures  seem  highly  desirable, 
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6.3  PROJECT  COMPARABILITY 

When  asked  what  project  differences  contribute  to  making  data  collection 
measures  lack  comparability,  project  managers  tended  to  check  a number  of 


reasons.  The  tallies  were: 

e Lack  of  standard  measures  62t 

e Lack  of  standard  techniques  57 

e Software  application  differences  57 

t Lack  of  standard  organization  52 

e Differences  In  management  practices  48 

e Technological  development  differences  43 

e Relative  "tightness"  of  schedules  05 

e Customer  requirements  differences  05 

. -r . ' • r ■ 


In  short,  project  managers  believe  that  projects  tend  to  differ  on  a 
complex  of  factors  rather  than  on  a single  aspect.  This  does  threaten  to 
make  methodological  research  difficult  by  making  It  hard  to  obtain  an 
adequate  sample  of  similar  projects  upon  which  to  base  an  assertion.  If 
enough  of  the  variables  are  "standard,"  It  Is  possible  to  select  one  or 
more  techniques  or  organizations  as  basic  variables  In  project  comparisons. 

When  asked  how  the  comparability  of  measures  taken  from  different  projects 
could  be  Improved,  project  managers  said: 

a Develop  quantitative  measures  of  project  characteristics 
e Steeiardlae  the  requlramants  of  the  development  process 
e Raise  and  broaden  the  sampling  level  (above  the  minor 
low-level  differences) 

e Use  Independent  audit  teams  across  projects 
e Circtaastances  differ  too  greatly  to  achieve  truly 
comparable  projects. 


'W. 


I 


asiriia^! 


28 


It 

Since  the  comparability  of  projects  rose  to  the  top  several  times  in  the 
survey,  it  appears  to  be  an  important  problem,  and  one  that  stymies  good 
estimates  of  costs,  quality  and  project  requirements.  However,  although 
the  suggested  solutions  appear  difficult  to  implement,  they  seem  within 
the  goals  of  the  repository. 

6.4  PROJECT  RESISTANCE 

The  reluctance  of  project  personnel  to  release  data  is  an  oft  mentioned 
problem  for  data  collection.  Project  members  resist  close  monitoring 
as  a perceived  threat  to  their  Independence  and  professional  competence 
.nd  managers  say  that  they  will  not  release  data  that  might  help  or  give 
comfort  to  competitors.  Less  than  a third  of  the  project  managers  In  the 
survey  checked  project  resistance  as  a data  collection  problem.  When 
asked  what  justification  was  given  for  project  resistance,  the  project 
monitors  said: 


Interference  with  main  task 

43X 

Collection  costs 

19 

Political  Consequence 

19 

"None  of  your  business"  attitude 

14 

Company  sensitive  Information 

00 

The  results  are  a little  surprising  In  that  the  objective  reasons  "Interference" 
and  "costs"  are  cited  and  the  reluctance  to  release  competative  Information 
Is  not  cited  at  all.  However,  the  perceived  threat  and  the  ways  to 
deal  with  It  were  rocognifod  In  the  swggostloRS  for  Improvement: 

0 Oo  not  penal lie  project 
0 Keep  data  sources  anonymous 

0 81 ve  pesttivo  assurance  that  honesty  1*  desired  and  that 

corrective  Ktlons  will  enhance  rather  than  hinder  project 
ptf  'activity 

0 Explain  the  purpose  of  the  data  collection 
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HoMever,  In  keeping  with  the  main  rationale  - Interference  with  main  task  - 
six  people  suggested  schedule  and  budgetary  recognition  of  the  data  collection 
task  and  making  adjustments  when  data  collection  did  Interfere  with  cost. 

6.5  COLLECTIOW  COSTS 

On  being  asked  where  the  principle  costs  of  collecting  software  development 


data  lay.  the 

e 

project  managers  said: 
Volume  of  data 

43t 

e 

Difficulty  of  measuring 

43 

a 

Number  of  measures  necessary 

38 

e 

Preparation  of  reports 

38 

0 

Interference  with  work 

33 

e 

Frequency  of  reporting 

24 

e 

Data  reduction  volume 

19 

e 

Lack  of  defined  go«1s 

D5 

Apparently  costliness  Is  due  to  a broad  spectrum  of  factors  rather  than  any 
Isolated  one.  The  amount,  variety  and  frequency  of  measuring  combine  with 
the  difficulty  and  Interference  effects  to  Inflate  costs  that  are  too  often 
bourne  by  technical  rather  than  support  funds  and  personnel . 

The  suggestions  advanced  for  reducing  costs  Include  fOur  recomendatlons  for 
automatic  collection,  three  far  standardization,  and  two  for  streamlining 
and  generalization.  One  person  rocoawandad  better  planning  and  one  said 
"Don't  collect  ahy**.  One  dissident  voice  suffostod: 

"They  tcosts]  probably  cahnot  be.  or  should  not  be.  reduced 
If  any  real  progress  Is  to  be  mads  In  utilizing  the  experi- 
ence of  oip  proje&tos  tha  heals  for  oxpecutlons  on  another. 

In  any  case,  the  costs  of  data  collection  may  be  Justified 
In  the  probable  reduction  of  other  costs." 
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When  the  managers  wre  asked  if  they  thought  cost  estimates  were  inflated  as 
a resistance  tactic,  six  agreed  'yes'  and  five  said  'probably.'  To  some 
degree  then,  the  managers  agree  that  the  costs  of  data  collection  may  not  be 
as  great  as  alleged,  which  may  account  for  its  low  rank  as  a problem. 

6.6  VALIDITY 

Despite  the  low  number  of  persons  citing  invalid  measures  as  a major  problem 
for  data  collection,  quite  a few  persons  responded  to  the  question  asking  for 


the  major  sources  of  error.  These  responses  were: 

e Subjectivity  in  measurement  48% 

e Poorly  defined  parameters  43 

e Variability  from  time  to  time,  project  to  project  38 
e different  measures  project  to  project  33 

e Insensitive  measures  to  project  difficulties  29 
e Too  infrequent  for  adequate  control  14 


Although  subjectivity  ranks  high,  the  heavy  endorsement  given  to  the  next  three 
ranking  responses  indicates  that  standardization  and  consistency  of  the  measures 
collected  are  seen  as  problems  by  many  of  the  managers. 

Suggestions  to  improve  the  validity  of  measures  include: 

• Derive  and  enforce  standardization 

e Increase  prograamer  confidence  in  the  purposes  of 
data  collection 

e Better  use  of  automated  controls  to  assess  the  number 
of  compilations,  test  runs,  failures/successes,  etc; 
in  general,  bettor  instrumentation 

e Comprehensive  stu^y  of  each  project  with  respect  to 
the  data  to  be  collected  from  personnel  on  the  pro- 
ject and  define  standari  zed  measures  for  the  piejeet 

• Provide  (use  and  pay)  an  independent  audit  team 
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• Concurrence  on  consistent  definitions  for  netsures  deened 
pertinent,  plus  objective  tracking  end  reporting  of  those 
measures  on  a regular  basis  In  terms  that  do  not  presume 
an  Intimate  understanding  of  the  package  measured 


6.7  FALSIFICATION 

Good  response  «ms  also  received  to  the  Item  on  distorted  or  falsified  data 
despite  Its  low  rank  In  perceived  Importance.  In  response  to  the  question 
on  what  sort  of  distortions  occurred,  the  project  managers  said: 


e Unconscious  optimistic  or  pessimistic  bias  48S 
e Deemphasis  on  failures  48 
e Overemphasis  on  successes  33 
e Cosmetic  revision  In  summarization  29 
e Less  detail  In  summarization  24 
e Coverups  of  difficulties  09 
e Deliberate  falsifications  and  half  truths  05 
e Poor  classification  05 
e Unclear  definition  05 
e Not  considering  people's  reactions  05 


Most  Interesting  here  Is  the  rejection  of  dellhereu  falsification  of  data 
and  the  emphasis  on  unconscious  hies.  Distortions  ceueod  he  filtering  and 
suMarlzatlon  of  data  stand  at  midlevel . 


In  response  to  the  questions  on  hoM  managers  might  penetrate  or  detect  falsified 
or  distorted  data,  the  project  managers  suggested: 

"Through  technical  awareness  and  thorough  counter  and  cross 
checks  wherever  possible." 

"The  only  practical  way  is  to  be  part  of  the  project  and  know 
it  as  well  as  the  technical  people  - which  may  not  be  really 
practical ! " 

"Use  automatic  tools." 

"Track  performance  on  a periodic  basis  and  review  progress  with 
project  members." 

"Provide  IG  function  with  highly  knowledgeable  software 
experts  from  PHO  staff  (not  corporate  headquarters.)" 

"Allow  the  data  supplier  to  be  completely  free  to  provide  the 
data  without  reprimand  or  reward.  Cross-check  data  by  data 
collection  in  some  other  fashion." 

"Success  depends  upon  having  a very  clear  understanding  of  the 
need,  the  requirements,  and  user  biases.  Unless  a project  is 
aiming  at  clearly  delineated  performance  characteristics 
almost  any  assessment  is  equally  valid." 

"Know  what  you  are  doing.  If  you  do,  falsehoods  quickly 
stand  out." 

"Avoid  distortions  by  good  planning." 

Apparently  the  project  managers  place  their  faith  in  technical  competence 
reinforced  by  Independent  audits  and  cross-checks.  Recognition  is  given  to 
psychological  factors,  but  without  strong  emphasis. 
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7.  HAWTHORNE  EFFECTS 

It  has  been  alleged  that  the  very  act  of  measuring  performance  can  influence 
that  performance  favorably.  The  project  managers  were  asked  whether  they 
thought  that  data  collection  could  enhance  the  reliability  and  excellence  of 
project  work.  There  was  moderate  agreement  with  this  statement  according  to 
the  responses: 

e Encourages  taking  greater  care  43* 

e Makes  workers  aware  of  inadequacies  ^ 

e Makes  Job  seem  important;  gives  recognition  to  33 

workman 

e Makes  worker  act  more  deliberately  05 


8.  ^ TECHM0L061CAL  ADVANCE 


The  project  managers  were  asked  what  developments  they  saw  in  the  data- 
processlng  state-of-the-art  that  might  improve  or  hinder  the  process  of 
collecting  software  development  data.  They  said: 


HELP 

HINDER 

Automated  data  collection  system 

67% 

05”4 

Automated  management  toojs 

57 

00 

Greater  standardization  of  language  and  tools 

52 

05 

Improved  measurement  techniques 

43 

05 

Proofs  of  correctness  methodology 

24 

19 

Structured  programming 

24 

05 

Software  engineering  techniques 

24 

10 

More  analytic  devices  for  software  development 

24 

00 

Increased  emj^sis  on  reliability 

■ j . , • 
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In  short,  greatest  hope  is  seen  to  lie  in  greater  automation  and  better, 
standardized  measures.  Although  not  felt  generally,  both  proofs  of  correct- 
ness and  the  greater  emphasis  on  reliability  are  seen  as  adding  somewhat  to 
the  problems  of  data  collection. 

9.  SUMMARY 

In  a survey  consisting  largely  of  prograimning  project  managers  and  system  pro- 
gram offices,  a certain  polarization  of  opinion  might  be  expected.  However, 
there  is  a wide  spread  of  opinion  on  the  sensitivity  of  all  kinds  of  data 
items.  Most  sensitive  are  cost  data  and,  more  so,  data  that  directly  reflects 
project  performance  evaluations  such  as  cost  and  schedule  variances.  Least 
sensitive  are  objective  statistics  about  resources,  program  modules,  modifi- 
cations, problems  and  errors  that  do  not  reflect  cost  or  performance  evalua- 
tions. 

I 

In  reference  to  the  software  development  process,  the  managers  felt  that 
planning,  analysis,  and  review  in  the  early  developmental  phases  were  inade- 
quate and  that  insufficient  Information  exists  or  is  collected  about  these 
phases.  In  later  phases,  independent  test  teams  are  not  used  as  often  as  one 
might  desire  (only  30X  of  the  time)  and  data  is  not  often  collected  during 
the  operational  phase  of  a system.  Although  many  projects  (approximately  50%) 
were  governed  by  the  MIL-STD's  only  about  25%  thought  these  standards  were 
adequate.  Most  projects  (90%)  used  some  sort  of  progress  report  and  75%  used 
schedule  variance  for  project  control . No  other  data  were  used  by  more  than 

I 50%  of  the  projects.  Hence,  one  might  conclude  that  very  few  projects  "con- 

trol by  the  numbers"  from  which  It  might  be  concluded  that  most  project 
managers  control  their  projects  by  personal  supervision.  On  the  other  hand, 
the  dissatisfaction  of  the  managers  with  the  variations  in  data  standards  and 
procedures  from  project  to  project  may  make  them  distrustful  of  what  the 
collection  data  may  tell  them. 


